JavaScript Object Notation (JSON)

• Jedes gültige JSON-Dokument soll ein gültiges JavaScript sein
• JSON wird zur Übertragung und zum Speichern von strukturierten Daten eingesetzt
• Insbesondere bei Webanwendungen und mobilen Apps wird es in Verbindung mit JavaScript, Ajax oder WebSockets zum Transfer von Daten zwischen dem Client und dem Server häufig genutzt.

Das GeoJSON Format

• GeoJSON ist ein offenes Format um geografische Daten nach der Simple-Feature-Access-Spezifikation zu repräsentieren.
• Dafür wird die JavaScript Object Notation verwendet.

Die Struktur der Daten kann man sich mit einem JSON Viewer anschauen

Download von Beispieldaten

• Overpass Turbo kann verwendet werden um Beispieldaten zu bekommen

https://overpass-turbo.eu/

Exkurs OpenStreetMap Daten

• Auf Overpass Turbo können Daten für Map Features exportiert werden
• Eine Liste der erhältlichen Map Features gibt es auf http://wiki.openstreetmap.org/wiki/DE:Map_Features

Beispiele für GeoJSON

Das Paket jsonlite

install.packages("jsonlite")

library(jsonlite)
citation("jsonlite")

## 
## To cite jsonlite in publications use:
## 
##   Jeroen Ooms (2014). The jsonlite Package: A Practical and
##   Consistent Mapping Between JSON Data and R Objects.
##   arXiv:1403.2805 [stat.CO] URL http://arxiv.org/abs/1403.2805.
## 
## A BibTeX entry for LaTeX users is
## 
##   @Article{,
##     title = {The jsonlite Package: A Practical and Consistent Mapping Between JSON Data and R Objects},
##     author = {Jeroen Ooms},
##     journal = {arXiv:1403.2805 [stat.CO]},
##     year = {2014},
##     url = {http://arxiv.org/abs/1403.2805},
##   }

JSON importieren

library("jsonlite")
DRINKWATER <- fromJSON("data/RomDrinkingWater.geojson")

names(DRINKWATER)[1:3]

## [1] "type"      "generator" "copyright"

names(DRINKWATER)[4:5]

## [1] "timestamp" "features"

Die Daten anschauen

head(DRINKWATER$features)

##      type             id properties.@id properties.amenity properties.flow
## 1 Feature node/246574149 node/246574149     drinking_water     push-button
## 2 Feature node/246574150 node/246574150     drinking_water            <NA>
## 3 Feature node/246574151 node/246574151     drinking_water            <NA>
## 4 Feature node/248743324 node/248743324     drinking_water            <NA>
## 5 Feature node/251773348 node/251773348     drinking_water            <NA>
## 6 Feature node/251773551 node/251773551     drinking_water            <NA>
##   properties.type properties.name properties.name:fr properties.wheelchair
## 1          nasone            <NA>               <NA>                  <NA>
## 2            <NA>            <NA>               <NA>                  <NA>
## 3            <NA>            <NA>               <NA>                  <NA>
## 4            <NA>            <NA>               <NA>                  <NA>
## 5          nasone            <NA>               <NA>                  <NA>
## 6            <NA>    Acqua Marcia        Eau potable                   yes
##   properties.created_by properties.indoor geometry.type
## 1                  <NA>              <NA>         Point
## 2                  <NA>              <NA>         Point
## 3                  <NA>              <NA>         Point
## 4                  <NA>              <NA>         Point
## 5                  <NA>              <NA>         Point
## 6                  <NA>              <NA>         Point
##   geometry.coordinates
## 1   12.49191, 41.89479
## 2   12.49095, 41.89489
## 3   12.48774, 41.89450
## 4   12.48773, 41.89354
## 5   12.48529, 41.88539
## 6   12.48386, 41.89332

Github JSON Daten

• Es lassen sich auch Dinge aus dem Web auslesen:

my_repos <- fromJSON("https://api.github.com/users/japhilko/repos")

names(my_repos)

##  [1] "id"                "name"              "full_name"        
##  [4] "owner"             "private"           "html_url"         
##  [7] "description"       "fork"              "url"              
## [10] "forks_url"         "keys_url"          "collaborators_url"
## [13] "teams_url"         "hooks_url"         "issue_events_url" 
## [16] "events_url"        "assignees_url"     "branches_url"     
## [19] "tags_url"          "blobs_url"         "git_tags_url"     
## [22] "git_refs_url"      "trees_url"         "statuses_url"     
## [25] "languages_url"     "stargazers_url"    "contributors_url" 
## [28] "subscribers_url"   "subscription_url"  "commits_url"      
## [31] "git_commits_url"   "comments_url"      "issue_comment_url"
## [34] "contents_url"      "compare_url"       "merges_url"       
## [37] "archive_url"       "downloads_url"     "issues_url"       
## [40] "pulls_url"         "milestones_url"    "notifications_url"
## [43] "labels_url"        "releases_url"      "deployments_url"  
## [46] "created_at"        "updated_at"        "pushed_at"        
## [49] "git_url"           "ssh_url"           "clone_url"        
## [52] "svn_url"           "homepage"          "size"             
## [55] "stargazers_count"  "watchers_count"    "language"         
## [58] "has_issues"        "has_projects"      "has_downloads"    
## [61] "has_wiki"          "has_pages"         "forks_count"      
## [64] "mirror_url"        "open_issues_count" "forks"            
## [67] "open_issues"       "watchers"          "default_branch"

Weiteres Beispiel für JSON Daten

• Die Ergast Developer API ist ein experimenteller Web Service, der eine historische Aufzeichnung von Motorsportdaten liefert.

Ergast Daten lesen

library(jsonlite)
res <- fromJSON('http://ergast.com/api/f1/2004/1/results.json')
drivers <- res$MRData$RaceTable$Races$Results[[1]]$Driver
colnames(drivers)

## [1] "driverId"        "code"            "url"             "givenName"      
## [5] "familyName"      "dateOfBirth"     "nationality"     "permanentNumber"

Daten der New York Times

• Die New York Times hat mehrere APIs als Teil des NYT-Entwickler-Netzwerks.
• Es ist eine Schnittstelle zu Daten aus verschiedenen Abteilungen, wie Nachrichtenartikel, Buchbesprechungen, Immobilien, etc.
• Registrierung ist erforderlich (aber kostenlos) und ein Schlüssel kann hier erhalten werden.

New York Times Beispiel

article_key <- "&api-key=c2fede7bd9aea57c898f538e5ec0a1ee:6:68700045"
url <- "http://api.nytimes.com/svc/search/v2/articlesearch.json?q=obamacare+socialism"
req <- fromJSON(paste0(url, article_key))
articles <- req$response$docs
colnames(articles)

##  [1] "web_url"           "snippet"           "lead_paragraph"   
##  [4] "abstract"          "print_page"        "blog"             
##  [7] "source"            "multimedia"        "headline"         
## [10] "keywords"          "pub_date"          "document_type"    
## [13] "news_desk"         "section_name"      "subsection_name"  
## [16] "byline"            "type_of_material"  "_id"              
## [19] "word_count"        "slideshow_credits"

Das XML Paket

library(XML)
citation("XML")

## 
## To cite package 'XML' in publications use:
## 
##   Duncan Temple Lang and the CRAN Team (2016). XML: Tools for
##   Parsing and Generating XML Within R and S-Plus. R package
##   version 3.98-1.5. https://CRAN.R-project.org/package=XML
## 
## A BibTeX entry for LaTeX users is
## 
##   @Manual{,
##     title = {XML: Tools for Parsing and Generating XML Within R and S-Plus},
##     author = {Duncan Temple Lang and the CRAN Team},
##     year = {2016},
##     note = {R package version 3.98-1.5},
##     url = {https://CRAN.R-project.org/package=XML},
##   }
## 
## ATTENTION: This citation information has been auto-generated from
## the package DESCRIPTION file and may need manual editing, see
## 'help("citation")'.

Das xml2 Paket

install.packages("xml2")

library(xml2)
citation("xml2")

## 
## To cite package 'xml2' in publications use:
## 
##   Hadley Wickham and James Hester (2016). xml2: Parse XML. R
##   package version 1.0.0. https://CRAN.R-project.org/package=xml2
## 
## A BibTeX entry for LaTeX users is
## 
##   @Manual{,
##     title = {xml2: Parse XML},
##     author = {Hadley Wickham and James Hester},
##     year = {2016},
##     note = {R package version 1.0.0},
##     url = {https://CRAN.R-project.org/package=xml2},
##   }

Erstes Beispiel

url <- "http://api.openstreetmap.org/api/0.6/
relation/62422"

library(xml2)
BE <- xmlParse(url)

Administrative Grenzen Berlin

Das XML analysieren

• Tobi Bosede - Working with XML Data in R

xmltop = xmlRoot(BE)
class(xmltop)

## [1] "XMLInternalElementNode" "XMLInternalNode"       
## [3] "XMLAbstractNode"

xmlSize(xmltop)

## [1] 1

xmlSize(xmltop[[1]])

## [1] 328

Nutzung von Xpath

Xpath, the XML Path Language, is a query language for selecting nodes from an XML document.

xpathApply(BE,"//tag[@k = 'source:population']")

## [[1]]
## <tag k="source:population" v="http://www.statistik-berlin-brandenburg.de/Publikationen/Stat_Berichte/2010/SB_A1-1_A2-4_q01-10_BE.pdf 2010-10-01"/> 
## 
## attr(,"class")
## [1] "XMLNodeSet"

Node parsen

url2 <- "http://api.openstreetmap.org/api/0.6/node/2923760808"
RennesBa <- xmlParse(url2)

Way parsen

url3 <- "http://api.openstreetmap.org/api/0.6/way/72799743"
MadCalle <- xmlParse(url3)

The Overpass API

Logo Overpass API

The Overpass API is a read-only API that serves up custom selected parts of the OSM map data.

(http://wiki.openstreetmap.org/wiki/Overpass_API)

Wichtige Information

http://wiki.openstreetmap.org/wiki/Map_Features

osm map features

Beispiel: Nutzung der Overpass API

Spielplätze Mannheim

Export der Rohdaten

Export Rohdaten

Import von der Overpass API zu R

Link1 <- "http://www.overpass-api.de/api/interpreter?
data=[maxsize:1073741824][timeout:900];area[name=\""

library(XML)
place <- "Mannheim"
type_obj <- "node"
object <- "leisure=playground"

InfoList <- xmlParse(paste(Link1,place,"\"];",
type_obj,"(area)[",object,"];out;",sep=""))

XML Output

Spielplätze in Mannheim

Das Arbeiten mit XML Daten (xpath)

Die Liste der ID’s mit dem Wert playground:

node_id <- xpathApply(InfoList,
"//tag[@v= 'playground']/parent::node/@ id")
## node_id[[1]]

Erste node id

latitude und longitude bekommen

lat_x <- xpathApply(InfoList,
"//tag[@v= 'playground']/parent::node/@ lat")
# lat_x[[1]];lat_x[[2]]

lat_x <- xpathApply(InfoList,
"//tag[@v= 'playground']/parent::node/@ lon")

Latitude Koordinate

Paket auf Github

library(devtools)
install_github("Japhilko/gosmd")

library(gosmd)

## Loading required package: maptools

## Loading required package: sp

## Checking rgeos availability: TRUE

## Loading required package: RJSONIO

## 
## Attaching package: 'RJSONIO'

## The following objects are masked from 'package:jsonlite':
## 
##     fromJSON, toJSON

## Loading required package: stringr

pg_MA <- get_osm_nodes(object="leisure=playground",
                       "Mannheim")
info <- extract_osm_nodes(OSM.Data=pg_MA,
                          value="playground")

Ausschnitt der Ergebnisse

Link

• Tutorial zur Nutzung der Overpass API

• Vignette xml2

Mehr Beispiele, wie man mit XML Daten umgeht:

• Daten aus XML extrahieren

http://www.stat.berkeley.edu/~statcur/Workshop2/Presentations/XML.pdf

• Duncan Temple Lang - A Short Introduction to the XML package for R

http://www.omegahat.net/RSXML/shortIntro.pdf

Noch mehr Informationen

• Web Daten manipulieren

http://www.di.fc.ul.pt/~jpn/r/web/index.html#parsing-xml

• Tutorial zu xquery

http://www.w3schools.com/xml/xquery_intro.asp

• R und das Web (für Anfänger), Teil II: XML und R

http://giventhedata.blogspot.de/2012/06/r-and-web-for-beginners-part-ii-xml-in.html

• String Manipulation

http://gastonsanchez.com/Handling_and_Processing_Strings_in_R.pdf

Das R-Paket XML - Gaston Sanchez

library("XML")

Gaston Sanchez - Dataflow

Seine Arbeit sieht man hier.

Das Arbeiten mit XML Daten

Gaston Sanchez - Webdaten bekommen

Funktionen im XML Paket

Einzelne Objekte finden

<www.openstreetmap.org/export>

osm export

Beispiel: administrative Grenzen Berlin

Administrative Grenzen für Deutschland

url <- "http://api.openstreetmap.org/api/0.6/relation/62422"

BE <- xmlParse(url)

Administrative Grenzen Berlin

Das XML analysieren

• Tobi Bosede - Working with XML Data in R

xmltop = xmlRoot(BE)
class(xmltop)

## [1] "XMLInternalElementNode" "XMLInternalNode"       
## [3] "XMLAbstractNode"

xmlSize(xmltop)

## [1] 1

xmlSize(xmltop[[1]])

## [1] 328

Nutzung von Xpath

Xpath, the XML Path Language, is a query language for selecting nodes from an XML document.

xpathApply(BE,"//tag[@k = 'population']")

## [[1]]
## <tag k="population" v="3440441"/> 
## 
## attr(,"class")
## [1] "XMLNodeSet"

Quelle für die Bevölkerungsgröße

xpathApply(BE,"//tag[@k = 'source:population']")

## [[1]]
## <tag k="source:population" v="http://www.statistik-berlin-brandenburg.de/Publikationen/Stat_Berichte/2010/SB_A1-1_A2-4_q01-10_BE.pdf 2010-10-01"/> 
## 
## attr(,"class")
## [1] "XMLNodeSet"

-Statistik Berlin Brandenburg

Etwas überraschend:

xpathApply(BE,"//tag[@k = 'name:ta']")

## [[1]]
## <tag k="name:ta" v="<U+0BAA><U+0BC6><U+0BB0><U+0BCD><U+0BB2><U+0BBF><U+0BA9><U+0BCD>"/> 
## 
## attr(,"class")
## [1] "XMLNodeSet"

Geographische Region

region <- xpathApply(BE,
  "//tag[@k = 'geographical_region']")
# regular expressions
region[[1]]

## <tag k="geographical_region" v="Barnim;Berliner Urstromtal;Teltow;Nauener Platte"/>

<tag k="geographical_region" 
  v="Barnim;Berliner Urstromtal;
  Teltow;Nauener Platte"/>

Landkreis

Barnim

Weiteres Beispiel

url2<-"http://api.openstreetmap.org/api/0.6/node/25113879"
obj2<-xmlParse(url2)
obj_amenity<-xpathApply(obj2,"//tag[@k = 'amenity']")[[1]]
obj_amenity

## <tag k="amenity" v="university"/>

Wikipedia Artikel

xpathApply(obj2,"//tag[@k = 'wikipedia']")[[1]]

## <tag k="wikipedia" v="de:Universität Mannheim"/>

xpathApply(obj2,"//tag[@k = 'wheelchair']")[[1]]

xpathApply(obj2,"//tag[@k = 'name']")[[1]]

Das C und das A

url3<-"http://api.openstreetmap.org/api/0.6/node/303550876"
obj3 <- xmlParse(url3)
xpathApply(obj3,"//tag[@k = 'opening_hours']")[[1]]

## <tag k="opening_hours" v="Mo-Sa 09:00-20:00; Su,PH off"/>

Hin und weg

url4<-"http://api.openstreetmap.org/api/0.6/node/25439439"
obj4 <- xmlParse(url4)
xpathApply(obj4,"//tag[@k = 'railway:station_category']")[[1]]

## <tag k="railway:station_category" v="2"/>

• Wikipedia Artikel Bahnhofskategorien

Exkurs: Bahnhofskategorien

• rvest: Easily Harvest (Scrape) Web Pages

library(rvest)

## 
## Attaching package: 'rvest'

## The following object is masked from 'package:XML':
## 
##     xml

bhfkat<-read_html(
  "https://de.wikipedia.org/wiki/Bahnhofskategorie")
df_html_bhfkat<-html_table(
  html_nodes(bhfkat, "table")[[1]],fill = TRUE)

Bahnhofskategorien Übersicht

Nur fliegen ist schöner

url5<-"http://api.openstreetmap.org/api/0.6/way/162149882"
obj5<-xmlParse(url5)
xpathApply(obj5,"//tag[@k = 'name']")[[1]]

## <tag k="name" v="City-Airport Mannheim"/>

xpathApply(obj5,"//tag[@k = 'website']")[[1]]

## <tag k="website" v="http://www.flugplatz-mannheim.de/"/>

xpathApply(obj5,"//tag[@k = 'iata']")[[1]]

## <tag k="iata" v="MHG"/>

OSM Ausschnitte herunterladen

<www.openstreetmap.org/export>

osm export

Mehr Beispiele, wie man mit XML Daten umgeht:

• Deborah Nolan - Extracting data from XML

• Duncan Temple Lang - A Short Introduction to the XML package for R

Noch mehr Informationen

• Web Daten manipulieren

• Tutorial zu xquery

• R und das Web (für Anfänger), Teil II: XML und R

• String Manipulation

• Nutzung, Vor- und Nachteile OSM

• Forschungsprojekte im Zusammenhang mit OpenStreetMap

Referenzen

citation("XML")

## 
## To cite package 'XML' in publications use:
## 
##   Duncan Temple Lang and the CRAN Team (2016). XML: Tools for
##   Parsing and Generating XML Within R and S-Plus. R package
##   version 3.98-1.5. https://CRAN.R-project.org/package=XML
## 
## A BibTeX entry for LaTeX users is
## 
##   @Manual{,
##     title = {XML: Tools for Parsing and Generating XML Within R and S-Plus},
##     author = {Duncan Temple Lang and the CRAN Team},
##     year = {2016},
##     note = {R package version 3.98-1.5},
##     url = {https://CRAN.R-project.org/package=XML},
##   }
## 
## ATTENTION: This citation information has been auto-generated from
## the package DESCRIPTION file and may need manual editing, see
## 'help("citation")'.

Das neuere Paket

citation("xml2")

## 
## To cite package 'xml2' in publications use:
## 
##   Hadley Wickham and James Hester (2016). xml2: Parse XML. R
##   package version 1.0.0. https://CRAN.R-project.org/package=xml2
## 
## A BibTeX entry for LaTeX users is
## 
##   @Manual{,
##     title = {xml2: Parse XML},
##     author = {Hadley Wickham and James Hester},
##     year = {2016},
##     note = {R package version 1.0.0},
##     url = {https://CRAN.R-project.org/package=xml2},
##   }

Referenzen

citation("XML")

## 
## To cite package 'XML' in publications use:
## 
##   Duncan Temple Lang and the CRAN Team (2016). XML: Tools for
##   Parsing and Generating XML Within R and S-Plus. R package
##   version 3.98-1.5. https://CRAN.R-project.org/package=XML
## 
## A BibTeX entry for LaTeX users is
## 
##   @Manual{,
##     title = {XML: Tools for Parsing and Generating XML Within R and S-Plus},
##     author = {Duncan Temple Lang and the CRAN Team},
##     year = {2016},
##     note = {R package version 3.98-1.5},
##     url = {https://CRAN.R-project.org/package=XML},
##   }
## 
## ATTENTION: This citation information has been auto-generated from
## the package DESCRIPTION file and may need manual editing, see
## 'help("citation")'.

Links

• XML parsen - Stackoverflow

• Processing of GeoJson data in R